fix(agentex-ui): abort the upstream request when the BFF client disconnects - #383
Merged
deepthi-rao-scale merged 3 commits intoJul 31, 2026
Merged
Conversation
…nnects The `/api/agentex` catch-all proxy streams SSE through unbuffered but passes no signal to `fetch`, so when the browser goes away — closed tab, navigation, or the 300s Istio route timeout — undici keeps the upstream request open. Each orphan leaves an agentex handler blocked in a Redis XREAD, pinning one connection from that pod's pool until it is exhausted and agentex can no longer serve requests. There is no timeout on the call either, so an orphan never expires. Pass `req.signal`, which App Router aborts on client disconnect, and answer the resulting fetch rejection with a 499 so a departed client is not reported as an upstream failure. A fixed timeout would be wrong here — it would cut legitimate long-lived SSE streams. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses review feedback: `req.signal.aborted` alone can mask a genuine transport failure that happens to coincide with a client disconnect, since the flag is already set by the time the catch runs. Match on `error === req.signal.reason` instead. `fetch` rejects with the signal's abort reason itself, so identity separates the two cases without consulting the flag at all. Matching on `error.name === 'AbortError'` would not work here: Next's abort reason is a `ResponseAborted`, and the class name is minified in a production build, so both name checks fail and every disconnect would rethrow as a 500. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
anitaliuu
approved these changes
Jul 29, 2026
deepthi-rao-scale
approved these changes
Jul 29, 2026
deepthi-rao-scale
added a commit
that referenced
this pull request
Jul 31, 2026
…; stop deleting shared stream (#385) ## What was broken When you open a task in the UI, the backend keeps a live stream open to push updates. That stream had **no way to end itself** — it only stopped if the browser disconnected. So when a task finished, the backend just kept listening forever, holding one Redis connection each time. These pile up until the pool is empty, at which point the pod can't serve requests (and its health check fails, so it sits stuck until a manual restart). A second bug: when one viewer left, the code deleted the task's **shared** event stream — yanking it out from under everyone else watching the same task. ## The fix - **End the stream when the task is done.** The backend already gets a `task_updated` event when a task finishes, so the stream now closes as soon as it sees a terminal status. - **Fallback for the rare cases** the event can't cover (viewer connects after the task already finished, or the producer dies silently): a lightweight check on the existing keepalive interval closes the stream then too. - **Stop deleting the shared stream** on one viewer's exit — the Redis TTL already cleans it up on its own. ## Testing Two integration regression tests (stream self-ends on terminal task; one viewer disconnecting doesn't delete the shared stream). **Verified live (A/B against `main`)** — same steps both times: create a RUNNING task → open `GET /tasks/{id}/stream` → complete the task, while watching Redis `blocked_clients`. | | on task completion | Redis connection | |---|---|---| | **`main`** | stream keeps `XREAD`-looping; kept sending `:ping` 15s+ after completion and only ended when the client was force-disconnected | `blocked_clients` stayed `1` — released only on client disconnect (**zombie**) | | **this branch** | stream delivers the terminal `task_updated` and returns immediately (`Ending SSE stream …: received a terminal task_updated event`) | `blocked_clients` `1 → 0` instantly, no client disconnect needed | On `main` the reader logged `Reading messages from Redis stream …` every ~2s indefinitely after the task was done; on this branch it stops the moment the terminal event arrives. Same reproduction the incident described, and the connection is released instead of pinned. > Note: the unit/integration suite runs in CI; the tutorial-agent integration jobs were red due to a pre-existing missing-`pytest-asyncio` issue unrelated to this change (fixed separately in scale-agentex-python). Related: the UI-side half of this leak is #383. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- greptile_comment --> <h3>Greptile Summary</h3> This PR updates task SSE subscription lifecycle handling. - Ends subscriptions when a terminal update is received, authoritative task state becomes terminal, or the task disappears. - Replays buffered events and closes when connecting to an already-terminal task. - Publishes failure updates and centralizes terminal/non-terminal status classification. - Preserves shared Redis streams when individual subscribers disconnect. - Adds integration coverage for terminal shutdown, late connections, reclaimed keys, transient lookups, and shared-stream retention. <details><summary><h3>Confidence Score: 4/5</h3></summary> The PR should not merge until recoverable FAILED tasks no longer terminate subscriptions that must observe their subsequent RUNNING updates. The stream returns upon receiving any FAILED task update, while the task service intentionally supports forwarding that same FAILED task again and publishing a RUNNING recovery update; viewers whose subscriptions closed on FAILED cannot receive that recovery or subsequent events. **Files Needing Attention:** agentex/src/domain/entities/tasks.py, agentex/src/domain/use_cases/streams_use_case.py, agentex/src/domain/services/task_service.py </details> <details><summary><h3>Important Files Changed</h3></summary> | Filename | Overview | |----------|----------| | agentex/src/domain/entities/tasks.py | Introduces canonical terminal and non-terminal status sets shared by state transitions and SSE termination. | | agentex/src/domain/services/task_service.py | Publishes FAILED updates through the normal update path, while existing retry behavior can still restore FAILED tasks to RUNNING. | | agentex/src/domain/use_cases/streams_use_case.py | Adds event-driven and authoritative fallback termination and stops deleting the shared Redis topic, but the previously reported recoverable-FAILED lifecycle conflict remains. | | agentex/src/domain/use_cases/tasks_use_case.py | Reuses the canonical non-terminal status set for transitions to terminal states. | | agentex/tests/integration/test_task_stream.py | Adds regression coverage for terminal shutdown, late subscribers, vanished tasks, transient lookups, reclaimed keys, and shared-stream preservation. | </details> <details><summary><h3>Sequence Diagram</h3></summary> ```mermaid sequenceDiagram participant Client participant SSE as StreamsUseCase participant DB as Task Store participant Redis Client->>SSE: Subscribe to task stream SSE->>Redis: Snapshot stream cursor SSE->>DB: Read authoritative task state alt Task already terminal SSE->>Redis: Replay buffered events SSE-->>Client: Events, then close else Task is non-terminal loop Until terminal, missing, or disconnect SSE->>Redis: Read new events alt Terminal task_updated received SSE-->>Client: Terminal event SSE-->>Client: Close stream else Status-check interval elapsed SSE->>DB: Recheck task alt Terminal or missing SSE-->>Client: Close stream end end end end Note over SSE,Redis: Subscriber exit does not delete the shared stream ``` </details> <sub>Reviews (14): Last reviewed commit: ["fix(agentex): terminate SSE task streams..."](85dfe6b) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=48824816)</sub> <!-- /greptile_comment --> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lucyakoroleva
approved these changes
Jul 31, 2026
danielmillerp
approved these changes
Jul 31, 2026
deepthi-rao-scale
enabled auto-merge (squash)
July 31, 2026 17:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
/api/agentexcatch-all BFF proxy streams SSE through unbuffered but passes nosignaltofetch. When the browser goes away — closed tab, navigation, or the 300s Istio route timeout on the UI'svirtualService— undici keeps the upstream request open. There is no timeout on the call either, so an orphan never expires, and the browser'sEventSourcethen reconnects and opens a fresh one.Each orphan leaves an
agentexhandler blocked in a RedisXREAD, pinning one connection from that pod's redis-py pool (REDIS_MAX_CONNECTIONS, default 200). They accumulate until the pools are exhausted, at which pointagentexfails to serve new requests and only a restart recovers it.Reported on Mayo Nonprod (Slack thread). At saturation Redis showed 1,094
connected_clientswith 943blocked_clients, while each of the twoagentex-uipods held ~570 established sockets to theagentexClusterIP and served one inbound client apiece. Restarting onlyagentex-uidropped backend Redis connections from ~1,094 to 279 without touchingagentex, confirming the UI as the holder. Introduced in 0f07dc7 (#350).Fix
Pass
req.signal. App Router aborts it on client disconnect, which propagates through undici and destroys the upstream request — and makes the existing Istio route timeout an effective ceiling on upstream lifetime. A fixed timeout would be wrong here, since it would cut legitimate long-lived SSE streams.The abort also rejects the in-flight
fetch, so that case is answered with a 499 instead of propagating as a 500 — a departed client isn't an upstream failure. A genuine transport error still propagates exactly as before.Verification
Built this branch, pointed
AGENTEX_API_URLat a connection-counting SSE upstream, opened 5 streaming clients and killed them:mainshapeNo
ResponseAbortedstack traces in the server log, server healthy afterward.The main risk of adding a signal is truncating a response that is still streaming after the handler returns, so that was checked explicitly on a production build:
POSTwithduplex: 'half'echoed back intacttsc --noEmit,eslint --max-warnings=0, andprettier --checkare clean.Follow-ups for the Agents team (not this PR)
Two
agentexbackend issues from the same thread remain open:cleanup_streamin thefinallydeletes a task-wide stream (streams_use_case.py:194). The topic is keyed by task id alone, so it is shared by every viewer. Worth landing with or right after this PR: today thatfinallyalmost never runs because disconnects don't propagate, and this change makes it run on every tab close. If something looks like a regression from this PR, it is that.XREADevery ~2.1s. The naive fix is unsafe: the client's retry loop (custom-subscribe-task-state.tsx,while (!signal?.aborted)) treats a clean server-side stream end as "reconnect immediately" — no backoff, no error-counter increment — so ending the stream server-side alone would produce a reconnect storm. Needs a coordinated client change.Separately and unrelated to the leak:
AGENTEX_API_URLfalls back silently tohttp://localhost:5003, so a rename on either side yields a UI that can't reach the backend with no configuration error.🤖 Generated with Claude Code
Greptile Summary
This PR fixes a connection-pool exhaustion bug in the Next.js App Router BFF proxy: upstream
fetchcalls were made without a cancellation signal, so client disconnects (tab close, navigation, Istio route timeout) left orphaned requests alive, each pinning a Redis connection in the backend's pool until restart.signal: req.signalto the upstreamfetchso undici tears down the backend connection whenever the browser client disconnects or the Istio timeout fires.fetchin a try/catch and returns HTTP 499 for abort-triggered errors, using identity comparison againstreq.signal.reason(rather thanerror.name) to correctly distinguish a client-disconnect abort from a genuine transport failure that coincides with a disconnect — validated against Next.js'sResponseAbortedinternal abort path.Confidence Score: 5/5
Safe to merge — a one-file, focused fix that adds a cancellation signal and handles the resulting abort error correctly.
The change is minimal and well-reasoned: it adds
signal: req.signalto the upstream fetch and uses identity comparison againstreq.signal.reasonto discriminate a client-abort from a coincident transport failure. The PR author verified the behavior against a production Next.js 15 build, confirmed no stream truncation on 2 MB bodies, and the comment in the code accurately documents whyerror.namewould be wrong here. No other code paths are affected.Files Needing Attention: No files require special attention.
Important Files Changed
signal: req.signalto upstream fetch and catches abort errors with identity comparison againstreq.signal.reason; logic is correct and well-commented.Sequence Diagram
sequenceDiagram participant Browser participant NextBFF as Next.js BFF (route.ts) participant Upstream as Agentex Backend participant Redis Browser->>NextBFF: SSE request NextBFF->>Upstream: "fetch(target, { signal: req.signal, ... })" Upstream->>Redis: XREAD (blocking) alt Client disconnects (tab close / Istio timeout) Browser--xNextBFF: disconnect Note over NextBFF: req.signal aborted (ResponseAborted reason) NextBFF->>Upstream: AbortSignal propagated → undici cancels request Upstream--xRedis: connection released NextBFF-->>Browser: 499 (client already gone) else Genuine upstream transport error Upstream--xNextBFF: TypeError (ECONNREFUSED etc.) Note over NextBFF: error !== req.signal.reason → rethrow NextBFF-->>Browser: 500 else Normal SSE stream Redis-->>Upstream: stream events Upstream-->>NextBFF: streamed response body NextBFF-->>Browser: unbuffered SSE endReviews (3): Last reviewed commit: "Merge branch 'main' into erichwoo/bff-ab..." | Re-trigger Greptile